14. Additive and Multiplicative Attention

Which of the following are valid scoring methods for attention?

SOLUTION:
  • Concat/additive
  • Dot product
  • General

What's the intuition behind using dot product as a scoring method?

SOLUTION: The dot product of two vectors in word-embedding space is a measure of similarity between them